Graphical user interfaces have created a revolution in the field of computers. Instead of having to memorize arcane commands, the user can now manipulate software applications by controlling graphical objects representing their various functions and components. Conventional computer systems, especially computer systems using graphical user interface (GUI) systems, accept user input from one or more discrete input devices, such as a keyboard for entering text, and a pointing device, such as a mouse with one or more buttons, for operating the user interface. The keyboard and mouse interface provides for creation and modification of documents, spreadsheets, database fields, drawings, photos and the like. This GUI has proven effective, but a new step in the evolution of computing has revealed several drawbacks to existing GUIs.
Specifically, the introduction of pen-based computing devices has changed the way individuals view the GUI, and the manner in which users interact with their computers. Some pen-based computing devices, include a personal data assistant (PDA) or the Tablet PC. While there are a number of designs, a pen-based computing device is generally interacted by user by way of a physical writable surface and a writing implement. The writable surface may be a display surface or a writing pad. Rather than a standard keyboard or mouse interface, the navigation and control of the GUI occurs via the writing implement. While pen-based computing devices have been helpful, the transition to a computer-based inking arrangement may prove difficult indeed. This new approach to user interfaces has revealed problems and deficiencies in the traditional GUI described above. Examples of these problems will be discussed below.
One common use of computers and GUIs is to generate and edit electronic documents. These electronic documents can contain text (e.g., electronic word processors) and/or images (e.g., pictures), which are displayed on the user's screen for editing. However, in a pen-based computing environment, users may attempt to use text-based techniques for selecting objects. These text-based techniques are restrictive and can be counterintuitive for working in a pen-based computer environment.
The selection in text systems is based on the placement of an insertion point in the document. A mouse input device is used for pointing and selecting objects. For example, the user typically uses the mouse to move an onscreen pointer to the desired characters, and presses a button on the mouse for an insertion point. Then the user drags the pointer over characters to select them. The selection of the particular character may be reflected in a change in its appearance. For example, electronic word processing programs may display in black on a white background. The text may be arranged automatically in uniform rows of text across the user's screen, where the rows of text are assigned a predefined height based on user-defined settings (e.g., the use of 12 pt. font, the line spacing, etc.). Upon selecting these words the selected text may be given a white color, and the rectangular area inhabited by the text in the row may be given a black background that serves as a blocked selection highlight, identifying the selected text. The black blocked selection highlight occupies the entire row height, and serves to differentiate the selected text from the non-selected text.
Although this previous approach to highlighting text works in the restricted uniform, line-by-line environment of traditional word processors, this approach is undesirable in other environments that allow a greater degree of freedom movement and expression, such as pen-based computing devices. For example, in systems where the text is handwritten ink (e.g., on a personal data assistant using a touch-sensitive screen or a Tablet), the user is permitted to write text above, below, and at other angles to any such regimented lines. The lines are often not present on a writing surface. In the freeform environment, the ink is placed on the writing surface in non-uniform patterns and irregular patterns. Users may write and sketch on the same electronic page and/or annotate ink sketches. As a result, the handwritten ink characters and sketches are not in straight lines, even if the guide lines are present. The “blocked” approach discussed above does not allow for efficient selecting irregular non-uniform patterns of handwritten ink, non-ink graphical objects, and/or ink sketches. As a result, the blocked approach in a handwritten environment results in confusion as to what is actually selected, selecting the wrong objects, imprecision and inefficiency for the user.
Notwithstanding the drawbacks of “block selection” techniques of conventional word processing programs and the like, various tools have been developed to assist a user in selecting specific graphical objects for manipulation. For instances, some software applications allow a user to form a “rubber band” rectangle around one or more adjacent graphical objects. With this rubber band rectangle tool, a user activates the selection function, places a cursor at a first location, and then drags the cursor to a second location. As the cursor is being dragged, the tool forms a rectangular selection box with the first location of the cursor and the current location of the cursor as opposite vertices of the selection box. By continuing to drag the cursor, a user can change the dimensions of the selection box to encompass one or more graphical objects. When the user releases the selection function, the graphical objects completely encompassed by the selection box are selected for subsequent manipulation.
While this type of selection tool provides some ability for a user to select individual graphical objects, this tool has several drawbacks. First, the tool forms a single rectangular selection area. In many cases, a user may wish to select multiple graphical objects that cannot be bounded by a single rectangular selection box without that box also encompassing graphical objects that the user does not wish to select. This maybe a particularly significant drawback when the graphical objects are handwritten words is irregular pattern and a user may want to simultaneously select words forming sentences that do not always start or end beginning or end of a line. Additionally, the single rectangular approach goes around an object akin to a surround select approach. This approach can be counterintuitive for working in a pen-based computer environment and may create barriers to adoption of pen-based computing systems.
Second, in order to select a graphical object, the selection box must completely encompass that object. If a user is working quickly, the user may inadvertently form a selection box that does not entirely encompass a desired graphical object, even if the box borders hits the graphical object. When the user then releases the selection function, the desired object may not always be selected. Some alternate arrangements of this tool require the selection box to encompass only a designated center of a graphical object, but the user may still encircle a large portion of a desired object without actually selecting that object. Third, the nature of handwritten ink can render a rectangular selection tool problematic and ineffective. For example, handwritten ink characters can have ascenders and descenders from individual letters that may overlap on each other when the ink is on a particular page. As a result, the overlapping nature of the handwritten ink can make it difficult to select a particular line of handwriting with a rectangular selection tool. Thus, the use of this tool requires an inconvenient amount of precision from the user and can be inefficient.
In some existing systems, a drag selection approach, akin to holding down a mouse button and dragging to select text in a text editor may select large areas of blank space (i.e., white space) on the page. When this selected text is cut and pasted (using standard computer-based text editing techniques), the large volume of selected blank space may produce an unintended and surprising result. This result is counterintuitive to the average computer user because conventional text editing systems work differently. The unfamiliar operation of a pen-based system compared to known text based systems creates barriers to adoption of pen-based computing systems.
Accordingly, there is a need for a selection tool that will allow a user to conveniently select one or more graphical objects in their entirety, without requiring an inconvenient amount of precision from the user and provide a significant degree of freedom for selection. Further, there is a need for a selection tool that can be flexibly used to select a graphical objects that are not arranged in a straight line for freeform environments, and which indicates to the user when one or more objects actually are selected during the selection process.